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ABSTRACT 

Evaluation, as currently practiced in vocational 
education, may not contribute to program improvement; in fact, it may 
be counterproductive. Rather than asking how evaluations can be 
improved, it might be better to ask if a relationship does, in fact, 
exist between evaluation and program improvement and, if such a 
relationship exists, whfct new assumptions and approaches might 
improve that relationship. Current activities center around the 
provisions in the Vocational Education Amendments of 1976 and 
subsequent guidelines, which provided tor the assessment of a sample 
oi the student's enrolled in vocational education in quantitative 
terms related- to student achievement measures. Although the 
conventional notion is that if, Jirore resources were 'available for 
evaluation, the findings would be more interpretable for action 
alternatives , an alternative view suggests that the relationship ot . 
social problem solving to evaluation is not clear. Ordinary 
knowledge, social learning, and interactive problem solving, ( such as 
through public hearings) may, in reality, be the basis of social 
problem solving. Some alternative suggestions for the assumptions 
that underlie present vocational evaluation practices include (1) 
abandoning the pursuit of evaluation activities that are purported to 
contribute to improvement, but do not; and (2) focusing on content 
and learner decisions at the local level. Further educational 
research should be cpnducted apolit really , with the understanding 
that its findings will take at least a generation to be reflected in 
changed practices and should include much social research. (KC) 
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FOREWORD 



The use of evaluations is generally accepted today as £ means' to a spe- 
cific end — that end being program improvement, Yetaccording to the premise * 
of Jerry P. Walker's work, Evaluation and Program Improvement; in Vocational 
Education, M As evaluatio**±s currently configured and practiced in vocational 
education, there is little discernable evidence that it contributes to program 
improvement; in fact, in many cases evaluations may well be counterproductive 
to program improvement." , * f 

Offered in the spirit of constructive discussion by an author who favors 
the program improvement process, this publication's principal purpose is to 
motivate those responsible for voc ed evaluation or program improvement to - 
reexamine their assumptions about the processes./ With this in mind, Dr. 
Walker presents* a/ review of these assumptions undergirding evaluation activi- 
ties, along with alternative propositions to those assumptions. Having pre- • 
sented*a counterview of the conventional relationship between evaluation and 
program improvement, Dr. Walker goes on to present recommendations for con- 
structive change. 

This paper is one of ten interpretive papers produced during the fifth 
year of the' National Center's knowledge transformation program. The review 
and synthesis in each topic area is intended to communicate knowledge and sug- 
gest applications. Papers in the series should be of interest to all voca- 
tional educators including teachers, administrators, federal agency personnel, 

and r T" rs ' 

Th/ profession is .indebted to Dr. Jerry P. Walker for his scholarship in 
preparing his paper. Dr. Walker is Associate Director of the^ffice of ( 
Research, Development, and Training, itt, the Department of Industrial Science 
at Colorado State University, Fort Collins, Colorado. He was formerly Asso- 
ciate Director for Evaluation at the National Center for Research in Voca- 
tional Education. 

^ Dr. Yvonna Lincdln of the University of Kansas, Dr. James Greenan of tLhe 
University of Illinois, Dr. Peggy Stank of the Pennsylvania Department of I 
Education, Dr. Tim Wentling of the University of IlUrflois, and Dr ; William* 
Hull and Dr., Linda Lotto of the National Center foiJ^Research in Vocational 
Education contributed to the development of, the paper through their critical 
review of the manuscript. Staff on the project included Shelley Grieve, Dr. 
Judith Samuelson, and Dr. Jay Smink. Clarine Cotton and Ruth Nunley typed the 
manuscript, and Janet Ray served as word processor operator. Editorial 
assistance was provided by Janet Kiplinger of the Field Service* staff. 

. • Robert E. Taylor 
* * Executive Director * 

T The National Center fot Research 
in Vocational* Education 

V 
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EXECUTIVE SUMMARY 



In recent decades, evaluation has come into widespread use as a means for 
gathering data with which to*make informed decisions in program improvement ' 
effort's. As evaluation in vocational education is presently practiced, how- • » 
ever, little evidence can be found that it is contributing significantly ' to 4 
program improvement. Rather than asking how evaluations can be improved, we 
might better ask if a relationship does, in fact, exist between, evaluation arv4# 
program improvement, and if such a relationship does exist, what new assump-- 
tions and approaches might improve the relationship. 

t 

% Current activities center around the provisions in the Vocational Educa- * 
tion Amendments of 1976 (P.L. 94-482). Subsequent clarification of efforts, 
the 1977 guidelines and 1979 policy memorandum, provided for the assessment of 
a sample of the students enrplled in vocational education in quantitative 
terms related to student achievement measures while they are in school and by ' 
follow-up studies after they leave school., A^ciational study of* vocational^ o 
education indicates that while this legislation is designed to serve program 
decisions, the federal definitions of improvement are ibeing largely ignored ) 
and program improvement efforts are less t han'enc our aging. « 

The conventional notion is tKat if more resources were available to . 
increase the capacity to do evaluation, then findings, vould* have greater 
validity, and be more inter pretable into action alternatives, thereby 
increasing their utility for program improvement. An alternative view sug- 
gests that the relationship of social problem solving to evaluation is not 
totally clear. Ordinary knowledge, sociai learning, and interactive problem , 
'solving implemented through such procedures as "committee deliberations and 
public hearings may, in Reality K be the basis of social problem solving. 

A thoughtful examination of the assumptions that undergird present prac- 
tices is in, order if constructive change is to result. Some alternatives for 
consideration include (1) abandoning the pursuif of evaluation activities. that 
are purported to contribute to improvement but that, in fact, do not, and (2) 
focusing on content and learner decisions — their "r ightness" and "fairness** — 
with the locus of decision making at the local level and decisions being both 
specific and public. s 

Educational research should be expanded. It 'should be conducted, however, 
under conditions that (1) are as ' apolitical as possible, (2) anticipate its 
bindings to take at least a generation to be reflected in changed practices, 
and (3) insist upon as tnuch social reseal?e-h as possible. A gradual shift is 
called for from an attitude of certainty to humility, from disdain for theory 
to centrality of theory, from ^valuation to professional social inquiry, and 
from program improvement to social problem solving. It is from such a per- 
spective that professionals can begin to design and implement effective 
evaluations . 

xi. 



INTRODUCTION 

The purpose of this paj5er is to contribute .to the improvement of Voca- 
tional education by illuminating and examining the assumptions that ;undejr- 
gird. existing evaluation and decision-making practices. In recent decades, 
evaluation has come intb widespread use as a means for gathering data with 
which to make informed decisions as- efforts have increased to improve the 
delivery of services in all sorts of endeavors. Evaluative activities <*xi&t 
to support' program' improvement efforts. 

If improvement is a virtually self-evident purpose of evaluation, why is 
there a need to examine the relationship between vcJcatibnal education eval- 
uation and program improvement?^ The reason is this: As evaluation is 
presently practiced in vocational education, there is little discernable 
evidence that it contributes to program improvement; in fact, in many cases 
evaluations may well be counterproductive to program improvement . * 

Tty.s premise is offered in the spirit of constructive discussion in favor 
of program improvement.. Assertions that programs ^re, *"irfr fact, effectively 
' being improved through evaluation canno* be- accepted out^of hand if an exami- 
nation of the relationship between evaluation and program improvement is to be 
constructive. To do so might lead to the position that since programs are 
effectively being improved, all that is* really needed is to do more of the 
same — or to somehow do it a little better. Continuation- A -or worse, 
escalation — of cur rent, evaluation policies and practices'in vocational educa- 
tion could possibly^ lead to more harm than good, more ritual than action, mdre 
rhetoric than realism, am* — perhaps most importantly— to moxe resentment than 
respect'and shared commitment among the local, state, and federal levels of 
vocational -education leadership. " * 

. How can evaluations be improved so as to improve ypcational educatiofe more 
effectively? Two assumptions implicit in this question- follow. v * 

* \ . 

o A positive relationship exists between the* quality of evaluations and 
N the quality of program improvement clecisions; /* *, 

o Gonducting existing evaluations more, affectively will, therefore, result 
in more effective program improvement efforts. ' 

, * . . • , ■ • — , 

These two assumptions are subject to discussion, however. It is better, per- 
haps, to pose the following* questions-. * ' *- 

o Does & relationship exist between evaluation and program improvement? 

o If such a relationship exists, what new assumptions and approaches might 
improve the relationship? 



o What additional agenda might be considered for inquiry, policy, and 
• . 'practice in vocational education -to further ,*he understanding and 

pursuit of more effective evaluations and program improvement efforts? 

Informed answers to these questions -can rejsult only if the individual 
examining them is knowledgeable about tftte background of ^current evaluation 
policy and practice in vocational education. The most recent and clearest 
picture of vocational education's efforts in evaluation and program improve- 
ment is seen from the national perspective. It is only at tfhis level that we 
can gain some general grasp of the directions being taken toward program 
improvement and the natur.e of <he evaluations being" conduc ted . In fact, some 
analyses of evaluation in vocatiojjflj* education use the language of federal 
legislation as the single" templ^'tefor determining/ the merit of evaluation 
practices in vocational education. D^tta (1979) sees the purpose of the 1976 
Vocatipna^ Education Amendments (P.L. 94-r482) as being "to ensure that eval- 
uation data will be used f6r planning, improvement and redirection" (p. 49). 
The inference that without the federal provisions, such, data w(?uld not be so 
used is clear (ibid.). * 

r * 

These provisions, which may seem tyrannous to thosfe unfamiliar with the 
long history of only modestly interpr etable and minimally utilized voca- 
v{ tional education evaluations, might better be described as dfter<nined and 
forebearing. . . . The remarkedly prescriptive evaluation requirements of 
the amendments may be what is- needed, (p. 49) 

^ ' - 0 

Two specific provisions for ^valuations are contained in P.L. 94-482.* 

it. Each State shall, during the five-year period of the State plan, 
evaluate the effectiveness of each program within the State being 
assist-ed with funds aVailable under this Act; and the results of these 
evaluations shall be used to revise the State's programs, and. shall be 

, piade readily available to the State advisory council; and 

J$. Each State shall evaluate, by using data collected, wherever possible, 
by statistically valid sampling techniques, eacji such program within 
the State which purports to impart entry level job skills according to 
the extent to which program completers and leavers — 



•n 



„ i. 'find employment in occupations related to their training, and 

ii. are considered l/y f their employers to be well trained and prepared 
for empl oymen t . ; . * „ 



These two statements are messages from 'the U.S. Congress, which say; in 
effect: (1)^ evaluate alj. vocational education programs and use the evaluative 
da ta^o, improve them, and ,( 2 ) determine (and demonstrate to us) that, in fact, 
vocational educatlx^ students are getting and keeping jobs. They were opera- 
% -tionalized through the evaluation guidelines presented in figure 1. 



^Education Amendments 1976; P.L. No. 94-482, 90 Stat. 081,- Sec. 112 (b)(1). 



FIGURE 1 

« 

t % 

SPECIFICATIONS IN THE REGULATIONS OF EVALUATION BY STATE BOARD 

The state board shall, during the five-year period of the. state plan, 
evaluate in quantitative terms the effectiveness of each formally organized 
program or project supported by federal, state, and local funds. Thetee eval- 
uations shall be in terms of: ^ 

(a) Planning and operational processes, such as: 

(1) Quality and availability of instructional offerings; * 
(2*) Guidance, counseling, and placement and follow-mp services; 

(3) Capacity and Condition of facilities and equipment; 

(4) Employer participation in cooperative programs of vocational 
education; ** ^ 

(5) Teacher/pupil ratios; and 

(6) Teacher qualifications. V 

(b) Results, of students achievement as measured, for example, by; 

(1) Standard occupational*, prof iciency measures; * 
• (2) Criterion-referenced tests; and - v j 

(3) Other examinations of students' skills, knowledge, attitudes, and 
readiness for entering employment successf ully . 

(c) Results of student employment success as measured, for exaEfyle, by: 

(1) Rates of employment and unemployment ; 

(2) Wage rates; 9 

(3) Duration of employment; and 

(4) Employer satisfaction with performance of vocational education, 
students a*s compared with performance of persons who have not had 
vocational education. 

(d) The results of additional services, as measured by the suggested 
criteria under paragraphs (a), (b), and (c) of this section, that the 

state provides under the Act to these special populations: 

« , •* 

(1) Women; 
^ (2) Members of mifiority groups; 

(3) Handicapped persons; t 

(4) Disadvantaged persons; and 

(5) Persons of limited English-speaking ability. 



SOURCE: Federal Register, October 3, 1977, Sec 104* 402, 
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^ These guidelines provided for the assessment of virtually -all of the some 
17 million students enrolled in vocational education courses across the coun- 
try.' The assessment was to be completed "in quantitative terms" via various 
student achievement measures while they were in school and by one or more 
follow-up studies after they left school. Further, evaluation of each program 
and of planning and operational processes was required. Finally, the results 
of all additional services to special populations weye to be quantified. 

Although apparently specific, the guidelines were 'widely interpreted as 
being ^uite^ei^eral— as saying in essence: "Do something 'evaluative' for all 
the programs during this five-year legislative period." As a result of the 
confusion surrounding the legislation and guidelines, a policy memorandum was 
issued by the Bureau of Adult and Vocational Education, U.S. Office of Educa- 
tion, ,in April of 1979, two and one-half y^ars after the 1976 Amendments. 
This memorandum defined "program" and permitted states to evaluate a sample of 
programs during the five-year period — students within courses, courses within 
programs, programs within schools, schools within districts, or districts" 
within states.* Clearly, some national picture of vocational education wa^ to 
be presented as a result of these evaluative activities. 

Unfor tunately ^^aTthough these efforts were designed to serve program, 
improvement decisions, they may have little actual relationship to-program 
improvement efforts. An unprecedented national study of vocational education 
completed by the National Institute of Education (1977, 1979a, 1979b, 1980a, 
1980b, 1981a, 1981b, l^glc) reveals a' picture of program improvement in voca- 
tional education that is less^than encouraging. ..In essence, the study says 
that the federal definitions of improvement are being largely ignored by 
vocational education and that those evaluation activities and criteria wi^ch 
are ostensibly intended to serve program improvement decisions have little 
actual bearing on program' improvement. 

The study suggests, further, that there are two Versions of reality in 
existence. One is the version of reality communicated in the array of infor- 
mation exchanges among the federal, state, and local levels of vocational 
education programs. The other i;s the version of reality perceived by those 
who actually observe programs in action. These two realities apparently do 
not bear any resemblance to each other. The problem IjLes in the contrast 
between the assumptions about evaluation and program improvement that underjfte 
the first reality and the actual behavior that defines the second reality. 
The difficulty centers around the differences between rationalistic views of 
social change and professional social inquiry on the one hand and the reali- 
ties of social problem solving on the other. 

The conventional opinion is that if there were just more resources to 
increase the capacity to do evaluation, then the findings would have greater 
validity . Therefore they would be more inter pretable in terms of action 
alternatives, thus the utility of evaluations to program improvement would be 
greatly increased. This Motion can be displayed schematically as folloWs. 

< 



RESOURCES 



CAPACITY 



VALIDITY 



t 



INTERPRETABILITY 



LABILITY |- 



UTILITY 



T 



! 



PROGRAM IMPROVEMENT 
^ (SOCIAL PROBLEM 
* SOLVING) 



The counterview to this conventional wisdom holds that the relationship of 
evaluation to social problem solving is, at best, ambiguous. Social problem 
solving necessarily and appropriately takes place in a political environment. 
(Certainly, all of fcjiat part of social problem solving tha\: is program 
improvement in vocational education takes pla.ce in a political environment.) * 
That environment is value laden; a multiplicity of values exists; the values 
are fluid and contradictory; ordinary knowledge is authocitative knowledge; 
the decision-making groups ane diffuse; convergence of opinion is sought; 
problems are without clear definition; and social interaction and' social 
learning are themselves the major problem-solving strategies. 

These conditions confraa^ with the assumptions associated with traditional 
evaluations; the stance is/value free and disinterested, or if value based, 
the focus is on the stated program goals; the criteria or "dependent vari- 
ables" are few, and they are unambivalent; authoritative knowledge is based on 
scientific conclusiveness; divergence in the implications from findings is 
more likely than convergence; t*he problem is well delimited; and analysis is 
the major problem-solving strategy advocated. Given this view, the schematic 
is as follows. 



RESOURCES 



t 



CAPACITY 



VALIDITY 



INTERPRETABILITY 



UTILITY 



PROGRAM IMPROVEMENT 
(SOCIAL PROBLEM 
SOLVING) 



The differences be.tween the fifst and the second views are fundamentally 
important in their implication regarding what to do about evaluation and 
program improvement. The solution g^ven the first vi£w is predictable. For 
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example, when state directors of vocational education were asked, "What fac- 
tors hampered the development of evaluation activity in your state?" the most 
frequently cited reason (over 50 percent) was^Tnadequate resources" (Wentling 
1980 ci,ted in Haney 1980, p. 27). A less predictable and less orthodox 
response from the viewpoint of the second position, is seen in the summary 
recommendation to evaluators of fered' by Lindblom and Cohen (1979). • 

To believe, as a thoughtful person might, that societies can make sig- 
nificant headway against' social problems does not imply that they can do 
so through PSI [professional social inquiryj. For, as we have suggested, 
much of the world's work of problem solving is accomplished not through 
PSI but through ordinary knowledge, through social learning, and through 
interactive problem solving, (p. 8) 



THfe problem with these two views is that they cannot be joined by accom- 
modation, compromise, or even fiat. It is ^ither basically true that the 
evaluations conducted in vocational education hold pVomise for program 
improvement if we pan but do more of them and do them better, or it is true « 
that the realities of social problem solving, as they now are and as they will 
remain, are such that their relationship to current evaluation practices 
remains, at best, sputious. 

The closest one might come to accepting both views is to suggest a scen- 
ario that might set in motion a slow but cumulative series of incremental 
shifts in social problem-solving structures in vocational education. TheSe 
shifts would recognize and build on the realities of social problem solving 
which are, at many points, antitheses to evaluation premises and practices. 
Eventually, a situation may emerge in which evaluations become a natural part 
of these processes. Such evaluations, however, Tnay be quite ali.ert to those 
processes that today's assumptions about professional social inquiry permit us 
to imagine. £uch a scenario will be discussed in a section to follow. First, 
however, there is a necessity to, define those propositions about social prob- 
lem solving and professional social, inquiry that help explain their indepen- 
dence from, or even antagonism to, each other. 
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ISSUES IN EVALUATION, 



A need exists to demonstrate the difficulties of actually conducting 
evaluations— difficulties that stem from the limited capacity of vocational ' 
education to do so and the unlikely validity of the findings even if capacity 
posed no problem. The more important task, however," is to explain as fully as 
possible why the paramount obstacles to the relationship between evaluation 
and program improvement result from inabilities to interpret and act on» eval- 
uative information, whatever its validity may be. 

Thus, in the sections to follow, each of the four conditions of capacity, , 
validity , interpretability , and utility are examined in terms of the assump- 
tions that one must accept in order to have confidence that the condition is 
likely to exist. In the case of tenuous assumptions, counterassumptions are 
offered in the form of propositions about that condition. 

i 



Capacity 

Assumptions that one must accept in order to have confidence jh^the capa- 
city of vocational education to conduct sound evaluations follow?" 

It Local school districts hpve a -sufficient number of competent eval- 
uators who are available to design evaluations, collect and analyze 
data, and provide evaluative findings about the needs, processes, and 
products of vocational education programs. * } 

2. Sufficient resources are available to local vocational educationists 
to enable them to acquire, maintain, and update the facilities, mate- 
rials,' and equipment necessary to conduct sound evaluations. 

The accuracy of these two assumptions is not certain. 

A comparison of vocational education evaluation requirements and resources 
with those 6f Title I of the Elementary and Secondary Education Act (Disadvan- 
taged) and of- the Education for All Handicapped Children Act (P.L. 94-142), 
suggests that (1) the requirements for vocational education are more t exten- 
sive, (2) there is less likelihood of the results being useful for program 
improvement purposes, and (3) the resources and assistance for conducting the 
evaluations" are more limited, Haney (1980) summarizes his views oft this point 
as follows: 

In sun*, -then, it seems to me that there are several different grounds for 
concluding that the current federal mandate for evaluating vocational 
education is overly ambitious. . . . Current requirements impose both 
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reporting and financial burdens on agencies implementing vocational educa- 
tion programs with federal support. . . . The requirements for vocational 
education are far more extensive, [than those for Title I and P.L. 94- 
142] . . . yet at the same time far less fully supported either in terms 
of funding or technical assistance.- . . . The major purpose of vocational 
education evaluation is program improvement at state, local, and institu- 
tional levels. Nonetheless, several of the evaluation criteria currently 
mandated for vocational education are of questionable utility in serving * 
that end. (p. 40) * « 

No one really knows how much evaluations in vocational education actually 
cost, and ^certainly not how much they should t cost. Even though.it has been 
argued earlier that an insufficient capacity to conduct evaluations has little 
bearing on whether programs are improved, the following proposition is offered 
\ as an alternative to the two assumptions just stated; ' 

o Proposition 1 . It is- unreasonable to expect local education agencies 
to have the resources and expertise necessary to conduct so\ind evalua- 
tions of their vocational education programs. Insisting that they 
attempt this despite their inability to do so ensures that most of the' 
subsequent evaluative findings will be inadequate and, perhaps*, 
suspect. , 



Validity 

X Validity refers to the extent to which the findings from professional 
social inquiry (or evaluation) are likely t:o be true. More technically, it is 
defined as the extent to which findings can be defended against the challenges 
of rival hypotheses. Two assumptions necessary in order . for us to accept the 
validity of vocational education^ evaluation findings are these. 

1. > Inquiries that purport to assess the consequence^ of vocational 

education can attribute those consequences to vocational education 
courses and programs. 

'2. Inquiries that purport to demonstrate the effectiveness of vocational 
education processes can demonstrate the consequences accruing from 
those processes. 

In both of these assumptions, the requirement for Attribution is clea*. 
In the first assumption, a relationship is called for between necessary 
antecedents and conditions. In the second, the relationship between antece- 
dents and necessafry consequences is required. While it is true that in many 
cases logic, not evidence, can be called upon to establish these relation- 
ships, it is also true that when rival interpretations are at le^st as, "plaus- 
ible as the' logic or evidence cited to demonstrate the relationship, 'the 
validity of the findings is called into question. Consider the following 
examples of "impact findings" relating to the first assumption. 

o Ninety percent of vocational education students find jobs related to 
their training. 

/ 
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o The average wages of former vocational education students are 30 per- 
cent higher than those of general education students* 

, o Most employers are very satisfied (60 percent) or quite satisfied (30 
percent) with the work performed by their employees who were vocational 
education students* 

Each of 'these is a positive statement about vocational education's impact. In 
the absence of evidence to the contrary, however, there is little reason to 
reject hypotheses such as the one that suggests that equivalent percentages of- 
nonvocational students are also employed, well paid, and viewed by their 
employers as satisfactory workers. v 

In the extreme, it is impossible to reject all possible* rival views 
because to da so would necessitate true experimental conditions in which all 
threats to internal and external validity are controlled. Because there are 
virtually no precedents in educational evaluation for the use of such designs 
(in which nontrivial consequences are studied), the practical issue is 
deciding on the point a"bove or below which the conduct of studies of conse- 
quences (oftpa termed impact or product evaluations) is worth it in terms of " 
their validity trade-offs. 



This same argument can be.appl 
these cases, if the intent is to d 
some educational process, then the 
"external standard of effectiveness 
logic. For examplel assume that a 
strates such facts ks these: 100 
facilities and -equipment are state 
job analyses. None |of these facts 
program effectiveness. 



led to process or formative evaluations. In 
emonstrate the relative effectiveness of 
relationship between that process and an 
must be well established by evidence for 
program review (evaluation) clearly demon- 
percent of the teachers* are certified, the 
of the art, or all programs are based on 
necessarily bears any relationship to 



One of the subtle but important problems related to program reviews in 
vocational education is that they produce tautological truths: Programs are 
good if all the stated criteria are met, and good programs are those that meet 
all of the specified criteria. The fact that some forrt of program review is 
the most common and accepted type of evaluatibn in vocational education makes 
this an even more difficult ^problem. The wide range of processes typically 
evaluated through some combination'of consultant .teams and self-evaluations by 
local districts is apparent in the following lisS'of separate evaluation forms 
available from the American Vocational Association for "assessing vocational 
education institutions (Wentling and Lawson 1975, p. 24). 



Distinguishing Characteristics 
Philosophy and Objectives 
Matching Objectives to Need 
Indicators of Success 
Delivery Systemd 
Organizations; Governing Body 
Administration 
Staff 

Finance and Business Management 
General Advisory Committee ' 



k f Planning 

Research and Evaluation 

Recruitment and Admissions 

Guidance and Counseling 
** Placement arid Follow-up 

Student Educational Records 

Student Activities 

Learning Resources 

Physical Plant 
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The use of various checklists, standards, and criteria is ccrtnmon in many 
applications of professional social inquiry. But the prevalence of their use 
does not justify their worth. Braybrooke and Lindblom (1963)ymake a strong 
argument against such approaches in these views: 

One primitive approach that might be calle<}."the naive criteria method" 
holds that merely announcing a few general vv^tues--"security," "employ- 
ment," and "price stability" perhaps — supplies* enough evaluative machinery 
to propel descriptive knowledge toward definite recommendations. . . . * 

J Besides being an inadequate method in practice, the naive criteria method 

is, of course, defective in conception, for it gives no clue as to the 

source, history, or relevance of the values postulated. They may, for all 
the method has to say, be invented on the spot. (pp. 6-7) 

While th^se comments may be a bit overstated with regard to educational* eval- 
uations, they do call into question those assumptions that necessitate the 
attribution of antecedents or consequences as requirements for validity. This 
proposition is » offered as an alternative to the two assumptions stated 
earlier: 

* 

0 Proposition 2 . The conditions required for 4 designing and implementing 
"the experimentally controlled evaluations necessary to determine voca- 
tional education's unique effects cannot be met or even approximated 
at state or local levels. The .degree Qf f equivocality of those*evalua- 
tion findings, which cannot approximate these^Qnditioti,s , makes such 
evaluations counterproductive to improvementor accountability 
purposes. # 9 

Two additional key assumptions must t>e accepted if one has confidence in 
tjie validity of evaluation findings. 

3. The individuals or agencies conducting the evaluations have n<? vested 
interests in whether the findings are positive or negative. 

9 

A. The units of observation and analysis can be unambiguously defined, 
and the observations can be reliably replicated. 

Some doubt may e&ist as to vhether the individuals charged with evaluating 
vocational education programs are entirely free of vested interests. Program 
review teams consist almost entirely of vocational educators. In fact, com- 
monly used rotational strategies for selecting evaluation teams make it likely 
that evaluator and evaluatee roles will be reversed at some point in time. 

State evaluation personnel report to a state director of vocational 
education — an individual who is required annually to provide quantitative 
findings tha£ demonstrate the extent of ^ogress toward program improvement 
goals in the state according to previously mandated and submitted plans. 
State advisory committees are charged with (and selected ija large part on the 
basis of their) advocacy of vocational education. , But they are also required 
to serve an independent evaluator role in their review of state plans and f 
accountability reports. 



The National Advisory Commii>tee for Vocational Education must also fill 
similarly conflicting roles* The U.S. Department of Education, in turn, is 
responsible for monitoring evaluation activities and at the same time is 
responsible to the U«S. Congress for demonstrating that federal funds hajve 
been well spent. Finally, many of the indepepdfcnt agencies that conduct 
studies of vocational education's ef fectiveness|\are, themselves, largely 
dependent on the U.S. Department of Education for continued funding. 

The point of the discussion of various roles and evaluation respon- 
sibilities is this: Virtually all groups involved in evaluating vocational 
education have other responsibilities that are inconsistent with those of a 
disinterested evaluator . Under these circumstances, it is unreasonable to 
expect validity standards will not be tempered by these other responsibili- 
ties. Thus, this proposition is offered concerning evaluation independence 
with regard to the validity of vocational education findings: 

o Proposition 3 . Reports of studies of vocational education's progress, 
or effectiveness conducted by agencies having a responsibility to fur- 
. ther the interests of public vocational education should be perceived in 
the same manner as other advocacy statements from any -agency that 
reports on its own progress and accomplishments. 

Several important problems are encountered in considering the assumption 
regarding clarity of terms (unambiguous definition) and replicability of 
studies. Any tesearch that purports to demonstrate that something has hap- 
pened to someone must specify both what has happened and to whom. Typically 
referred to as "treatments" and "subjects" in experimental research, these two 
entities are usually termed "vocational education program" and "vocational 
•education student" iir evaluations of vocational education. 

TJie problem in attempting to define^ these terms is that of meeting*\:he 
research requirements of either homogeneity or known variation. In the case 
of homogeneity, ^one must assume that the vocational education students being 
investigated have had an Equivalent exposure to the sam6 "treatment" or 
program and that, the differences among the students are not likely to inde- 
pendently account for what they 4id or did not learn or accomplish.. }f 
homogeneity cannot be reasonably assumed, then the differences in both stu- 
dents and programs must be ascertained and accounted for, in the design^ and 
statistical analyses. , * 

It should be noted that these definitional points are not the issues that 
can always be cited in contrasting the realities of educational evaluation 
conditions with those of experimental research. Instead, they are virtually 
insurmountable obstacles to any practical hope of relating evaluation findings, 
to program improvement decisions. This is true even if one were tq make the 
nearly unimaginable assumption that scientific evidence is the principal basis 
for program improvement decisions in vocational education, in any area of 
public education, or — for that matter — in nearly any social problem-solving 
context. Aside from this issue — to which we will turn in the section on 
interpretatility ^nd utility — the definitional problems themselves are high- 
lighted by observations such as the following: 
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•o Nationally, less than bne half of all vocational enrollees at the 
secondary level are in occupational preparation program^. (National 
Institute 1980b, p. VI-7) 

o There are not uniform rules for establishing how many courses, class i 
meetings or classhours per week constitute a program [for vocational / 
education]. (National Institute* 1980b, p. VI-5). 

v • 

o All previous studies have suffered from the inadequacy of cost and 
benefit data available for vocational education. ... A more rational 

* course . . . must be . . « gui?ded by agreed upon objectives and defini- 

* tions of output to measure these objectives. (Hu and Stromsdorfer 
1979, p. 214) 

o Users of placement data/should be very cautious when using placement 
data to make comparisons within and among spates because of the numerous 
ways in which key terms are defined. (McKinney et al. 1978, p. 14) 

The diversity and ambiguity of definitions are such that £ "program com- 
pleter" could be an individual who — % 
* * 

o spent less than one hour per day in a vocational course (a group that 
could include up to 90 percent of high school students according to 
a national survey completed in )L972); (National Institute 1980, 
p. VI-5) V 

o was eyrolledHLn a separate vocational education institution for all of 
his or her junior and senior years of high school (as in the pattern 
with joint vocational school students in Ohio, for example); 

o took a few courses in nonoccupational home economies' 

^> o completed an associate's degree program in a post^secondary institution 

and may have had over four years of continual training in a specific 
occupation; 

o completed a two- to three-day M quick-£tar t" tr aining <pr ogram; or^ 

o took a couplie of high school vocational courses as an adjunct to 
avocational interests ranging from motorcycles to computers. 

Nothing is necessarily wrong with the diversity of student purposes nor 
with the wide arrays in intensity, duration, and goals of vocational education 
programs. But there is something .wrong with evaluations that are unable to 
discern these variations and explain their effects. The problem is that for 
Whatever purpose such evaluations might be conducted, there can be no contri- 
bution to that pdrpose* For example, if the purpose of the evaluation is sim- 
ply compliance, then valuable time and dollars are wasted in collecting data; 
an estimate would suffice. If, on t t he other hand, the purpose is Account- 
ability, the inability to account for the relationships between program and 
outcome variables is likely to lead* more to a lacl^ of confidence than to con- 
fidence. Finally, if the purpose is program improvement, the inability to 
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define "student" or "program" and the inability to demonstrate the relation- 
ship between them permit findings abojut either to become a springboard for a 
nearly infinite range of guesses about what or how to improve — guesses that 
might have more^ foundation if based on ordinary knowledge and intuition. 
These notions lead to the following proposition: ' *■ 

o Proposition 4 . If evaluations cannot demonstrate relationships between 
program characteristics and student outcomes, the chances of effective 
decisions being made in efforts to improve programs are essentially 
random. 

The foregoing propositions are based on the assumption that the limita- 
tions of the information base provided through professional social inquiry are 
'the principal obstacles to more effective social problem solving. This is'$not 
the assumptidn underlying this paper. Even if it were, formidable obstacles 
stand in the way of developing a resource. aqd talent base with sufficient 
capacity to provide valid findings. Also, the capacity problems notwith- 
standing, the threats to validity reflect the inherent contrasts between the 
realities of social programs — such as vocational education — and the require- 
ments of professional social inquiry to define and control (or to control for) 
those realities. 

* * V 

In advance of a discussion of the conditions of interpr etability and 
utility, it is appropriate to emphasize that- the primary question .under dis- 
cussion is the appropriateness of an inquiry paradigm to the realitifes t of. 
social problem solving. If the paradigm is to be questioned, the assumptions 
on which it is based must be examined. 

0 

Assumptions about the capacity to ^onduct evaluations and the validity of " ' 
evaluation findings have been so Examined. Even though the propositions posed 
cast doubt on both of these assumption^^he assumptions related to inter- 
pretability and utility can be ex amiiyra independent of views on, capacity and 
validity. The following discussion enters directly into the relationship 
between evaluative findings (even assuming their validity— ?thus, also fhe 
capacity to produce them) and the nature of social problem solving — or in this 
context — the nat*u^e of program improvement in vocational education. m 



Intetpretability and Utility 

Posed as positive assumptions about the worth of professional social 
inquiry to social problem solving, these two relate/d\conditions may be inter- 
preted as follows: * * ^ 

o Inter pra£ability : Evaluative findings will help decisionmakers in 
selecting alternative actions fpr improving programs. 

o Utility : The actual behaviors that constitute social problem solving 
are influenced by the findings from professional social inquiry. 

If the results of professional social inquiry are to affect social problem, 
solving positively, they should point to certain action alternatives as mote 
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viable than others. If not,. then they have no interpretability and are simply 
statements of facts (assuming their validity) that have no meaning in terms of 
how programs might be improved. »The utility of findings is seen in terffi9» of 
how programs might be improved, as well as in the correspondence between eval- 
uations and subsequent program 'improvement behaviors. 

It is recognized that the possible relationships between evaluations and 
interpretability or utility are, at, best, infer entially determined. Urflite 
validity, interpretability and utility are not attributes of evaluatidns that 
can be judged independently. Instead, they'are possible consequences of eval-* 
uations,. their likelihood determined by the behaviors of decision-makers. 
Knowing how these groups think and behave as they pursue social problem 
solving is central tp an understanding of the interpretability and utility of 
the findings of evaluations or other forms of professional social inquiry. 

In order to put the assumptions held abdut the worth of. professional 
^social inquiry into context, the following excerpts are offered. These high- 
light the contrasts between those individuals who see inquiry as beneficial, 
even essential, and those who question its utility. Some optimistic views 
follow: 

* 

o There has been agreement, both within an$ without the ranks of educa- 
tion, that systematic inquiry has much to offer. m Indeed, there is 
agreement that massive, lasting changes in education cannot -safely be 
made except on the basis of objective inquiry . (Cronbach and Suppes 
1969, p.. 12) 

o In order for the public to make rational decisions concerning^investmen t 
in various education programs, comprehensive information about the costs 
and benefits of those programs is required. • • •' No vali,d policy 
statement- can , be made from analyses that do not include these com- 
"ponents, yet policy statements' axe made on the basis of incomplete 
information. .(Hu and Stromsdorfer 1979, p. 195) 



o Evaluation of programs will nearly always contribute to thje better ser- 
vice of students. Obviously, if evaluation offers both value judgments 
regarding what is happening within the program and consultative sugges- 
tions for improvement;, then ultimately t^os will have an effect on stu-* 
dents. The improvement of programs and offerings is the most important 
goal that evaluations can assume. (Wentling and Lawson 1975, p. 20) 

o Compared to other sources of knowledge, such as experience, authority, 
inductive reasoning, and deductive reasoning, application of the scien- 
tific method is undoubtedly the most efficient and reliableS (Gay 1981, 
P- 5) ^ , 

Clearly, opinions about the worth of inquiry 'are widespread. How^vjer, 
opposing viewpoints do exist. For the most part, their authorship is outside 
of education. A sampling of less optimistic statements about the role of 
inquiry as it relates to program improvement follows: 
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o To believe, as a thoughtful person might, that societies can make sig- 
nificant headway against social problems does not imply that they can do 
so through PSI [professional social inquiry]-. ... A realistic view of 
social problem solving stands in contrast to the rationalistic view * 
.implicit among the most enthusiastic advocates of FSI. (Lindblom and 
Cohen 1979, p. 91)* 

o As for the other major premise on which the utility of evaluation 
research is based, that policy makers will heed research results and 
respond by improving programming, there is not much positive evidence of 
either. We have notecU*jx>w the politics of program survival and the 
politics of higher policymaking accord evaluative evidence relatively 
minor weight in the (Jecisional^calculus . It is when evaluation results 
confirm what decision makers «ready believe or disclose what they are 
predisposed to accept that evaluation is most apt to get serious 
* attention. (Weiss 1979, p. 536) 

o In most educational organizations most of ^the time, goal w based planning 
models* are orderly facades that inhibit the management function of 
planning. (Clark 1980, p. 8) 

The contrasts between these „£wo sets of viewpoints are stark.. How'does one 
accommodate polar opposites? Let it suffice to say that reality is somewhere 
in between. ' 

Assumptions are paradoxical. Those most central to dur beliefs" are the 
ones we are least able to express. This is true of assumptions about .^valua- 
tipn and program improvement. Those that come closest to permitting B to 
examine the core relationship between evaluation and program improvemit are 
the least apparent 'and probably the most intractable. The following/lis t 
provides samples of the assumptions that our behavior as evaluators might 
imply: 

1. We live in orderly environments. There are causal conditions that 
give rise to patterns of consequences, 

2. The role of evaluation is to understand better the order inherent in 
the things we choose to examine. Thus, we seek "impact," "produces," 
and "outcomes" and try to establish the conditions that have caused 
them. 

3. We have rules for looking at and reporting reality, "Those rules 
emanate from the methods of science and are evident in our attempts to 
approximate reliable, valicf, and gener alifcable statements. 



5£ 

*The term goal-based planning models refers to "a comprehensive Information* 
base on which planning, can proceed, including data on internal and external 
impact" factors that will allow decision makers to assess alternative avenues 
of action." (Clark 1980, p. 2) 
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4. The core justification for our professional roles as evaluators is to 
provide information — consistent with the assumptions above — that ,will 
help others make better decisions ♦ * 

5. A princkpal justification for better decision making is that problems 
should be solved more effectively (or that program improvement should 
be strer%thene3) . 

6. Given ttte^bove assumptions, it follows that if we have sufficient 
capacity to conduct evaluations and if our findings are valid , they 
will be interpretable and have a utility that will be observable in 
improved programs. 

These assumptions ar^ apparently straightforward and logical. As previously 

noted, however, because so many uncontrolled (and uncontrollable) variables 

exist in social systems, the^final assumpt/ion gives pause* # ^ 

* * • * * 

As evidence accrues that evaluations do not have the hoped for and logi- 
cally expected utility, we tend to reconcile these assumptions with this 
observation in one of three ways. Probably the most common reaction is self- 
recrimination' leading to a "try harder" syndrome. This involves seeking to 
increase evaluative capacity and improve evaluative strategies so as to 
increase the validity of the findings. The logical and obstensibly sufficient 
goal of our efforts is increased capacity and validity. 

The widespread tepdency to react by trying harder is documented exten- 
sively. For exampl.e, following theit comment that only "deep objective 
inquiry" can lead to safe and lasting educational\change£, Cronbach and Suppes 
(1969) note that "inquiry* intfc educational matters is essential but extremely 
difficult" (p. 13). They continue with the assertion that "if scholarship of 
the same order [as that i*i medical research] ta^Lses in education, and is used 
with equal intelligence by practitioners, education in the twenty-first 
century will pull ahead in the race between civilization and catastrophe" 
(p. 271). 

Thfe second way people commonly react to evaluations with apparent lack of * 
utility is to attribute the problem to erroneous decision-malting strategies. 
The task becomes, at £he most. ambitious level, to enlighten the decision 
makers* or — a& the very least — to' anticipate and ct>pe with their errors. In 
either case, the frustration has to do with their "irrationality." Liridblom 
and Cohen (1979) remark as follows regarding this tendency: 

Perhaps the most familiar ' failures to achieve author! tativeness . . . stem 
ffom many irrational and nonrational human resistances to believing what 
... scientists generally say. # . . . Their general tenor is to lay ' 
blame on«users of PSI [professional social inquiry] and to suggest how the. 
practice of PSI, esecially the presentation of its results to potential 
users, has to be adapted to the shortcomings of the users, (p. 45) 

Stufflebeam afid associates (1971) highlight the "interface" role of the eval- 
uatpr as ; ai^ example' of the importance of presenting results to potential 
users. They offer this caution. 

16 
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The evaluation specialist must be particularly adroit in this interaction 
with the decision maker* The best service of the decision requests that 
all the logical alternatives be considered; however, to be effective the 
evaluation specialist can deal only With the alternatives that are real 
for the decision maker. ♦ ♦ ♦ This discussion does not mean 'that the 
evaluation specialist i6 a spineless "yes M man. Rather, it is a recogni- 
} tion that he must introduce new ideas into a decision maker's thinking in ^ 
manners that affect their consideration* (p. 298) 

Undergirding the "new ideas" that an evaluator should introduce to deci- 
sion makers is the utility of valid findings for improving decisions, making 
them more empirical, defensible, and most importantly — rational* In fact, the 
dichotomy is often^nade between "rational," in which decisions are made on the 
basis of hard evidence, and "nonrational," in which othier factors are con- 
sidered. "The decision maker who takes such other factors [e.g., political 
grounds] into account is not being irrational but is basing his [oft her] deci- 
sions on nonrational fac ttwe^^Stuf flebeam et al. 1971, p. 333). • 

Reactions of the second type are also seen in, what We^ss (1977) entitles — 
and challenges — as the" "conventional wisdom" approaches to increasing the 
utility of social research for public policymaking. 

Other mild and mapnerly reforms have been urged. Since the aim- has been 

to ^facilitate the transfer of knowledge, these reforms tend—to stress 

communication: 
\ 

computerized information systems that contain abstracts of research 

retrievable by key word; * 
increased personal contacts between researchers and policymakers to 

develop trust and communication; 
training for decision makers in social science mettod and theory; 
recruitment of more social scientists into positions of authority; 
greater willingness of researchers to interpret and promote their 

results; 

more critical syntheses of existing knowledge to ease the burden both 
of the search) for appropriate research and the assessment of its 
validity and appropriateness. 

So much for conventional wisdom, (p. 6-7) \j 

4 

The third tendency is to hold a view of social problem solving "devoid of 
any presupposition that it would necessarily be improved if aided by the 
effort of professional social inquiry. The contrasts between the assumptions 
upon which professional social inquiry is based and those that; may be inferred m 
from the behaviot evident in social problem solving must be made evident in 
or^Jer to do this. 




Propositions about Interpretability and Utility 

We turn now to some additional propositions. These are offered to help 
guide the activities of individuals seeking to use this third approach. 
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Pursuit of Authoritativeness 



The commonly held.assumptioft of professional social inquiry is that infor- 
mation is authoritative by virtue of its scientific conclusiveness. Further, 
it ought to be sufficiently authoritative to warrant direct action on the part 
of decision makers. The contrasting assumption is that such information — no 
matter what its validity or scientific conclusiveness — is at best dep^ndently 
authoritative and almost never independently authoritative. In other words, 
if the information does not correspond to and reinforce ordinary knowledge — in 
which case it is dependently authoritative — it cannot have an authoritative-- 
ness independent of this ordinary knowledge. 

Lindblom and Cohen (1979) suggest that suoh.misperceptions, on the part erf 
professional inquirers arise from a deeply rooted belief that scientific 
inquiry is the pursuit of verified propositions. They continue \s follows: 

Science is, in such a vi»ew, the pursuit of truth, a method of eliminating 
false opinion, a way* to perceive reality correctly. Even if they allow 
for persistent error and long-standing differences in findings, mqgt 
scientists and pPSlP [persons in professional social inquiry] find it dif- 
ficult to conceive of science and professional investigation as other than 
a process that ultimately moves toward ^convergence on propositions, toward 
an increasingly correct representation of reality. .All these notions seem 
to imply that pPSI should pursue confirmed 'knowledge or scientific conclu- 
siveness. And from the aspiration to achieve scientific conclusiveness, 
it is assumed that orie^can take a quick, short jump to authoritativeness. 
(pp. 44-45) ' ■ -jg. r . 

Other writers make similar points in ^tements such as the following: 

o The ultimate test.oj data acceptability -$.s political, . . . Rarely are 
data in their own right of such compelling force as to override their 
political significance. (Caplan 1977, p. 195) 

o We should not, and fortunately probably cannot, overwhelm the political 
judgment of the Congress with masses of factual data and technical 
analysis. (Dreyfuss 1977, p. 107) 

The proposition related to authoritativeness as it relates to vocational 
education follows: * 

o Proposition 5 . Seldom will evaluation findings that do not confirm 
their views be seen as authoritative by those responsible for* improving 
vocational education. Further, the utility of such findings is depen- 
dent oi> their correspondence with the ordinary knowledge that decision 
makers have of their environments. 



The Synoptic Ideal "* * 

Gross (197*9) presents a model of what has be*n called t^fie synoptic ideal 

of decision making. He suggests that the proposed model has the advantages of 
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defining all relevant variables, quantifying subjective information, and per- 
mitting all relevant information to be synthesized* The approach proposeS by^ 
Gross for vocational education is one that consists of "first describing the 
set of possible decisions that are available to the decision maker, second, 
quantifying the intrinsic worth or value of each possible decision in terms of 
a utility vfclue . . . and third, choosing that decision which possesses the 
highest utility value" (p. 111). 

The assumptions underlying such strategies and steps for decision making 
a*e questioned by Braybrooke and Lindblom (1963), when they explain the fail- 
ure in practice of such strategies as due to the "confusion between conceiv- 
ability and practicality, or between operability in principle and operability 
in practice" (p. 48). They continue by citing examples of the failure of the 
Synoptic ideal under eight separate propositions and arguing that the synoptic 
ideal is not adapted to the following situations: 

!• Man's limited problem-solving capacities 

2. The inadequacy of information , 

3. The costliness of analysis 

4. The failures in constructing a satisfactory evaluation method 

5. • The closeness of the observed, relationships between fact and value in 

policy making 

* 6. The openness of the systems of variables with which it contends 

7. The analyst' s" [decision maker 'sj need for strategic sequences of 
analytical moVes^ 

8. *The diverse forms in which policy problems actually arise (pp. 48-54) 

short, they asfffert that synoptic approaches simply have not and will not 
j*ork. In fact, they define the synoptic ideal in terms of £he attributes its 
advocates give it rather than by describing instances in which it has been 
actually used — because no such instances can be found. 

Some evaluation efforts in vocational education apparently accept the 
assumptions implicit in a synoptic decision-making strategy. For example, 
many of the features of the 1976 Amendments to the Vocational Education Act 
(P.L. 94-482) appear to support a synoptic approach. If th$ legislation and 
subsequent guidelines 'and policy directives were fully implemented, the U.S. 
Congress, federal administrative offices, and state and JLscal education agen- 
cies would now be able to we^Lgh the following kinds of fevilence in their 
deliberations about how vocational education can best be?*"lmproved to serve the 
public interest . 

\ 0 / 

o Summaries of several million records of students' achievement both in 
school &nd in their initial jobs 

o Data from their employers about whether these students were well trained 
and prepared for employment ^ 

o Quantified evaluations of the extent and quality of each vocational 
education program in each state on dimensions such as — 
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- planning and operational proc< 

- student achievement 

- student employment success 

- results of special services to women, minorities, handicapped and 
disadvantaged individuals, and those with limited English-speaking 
abilities ' ^ 

o Management evaluation reviews for compliance and quality (MERC-Q) of all 
states and territories 

o Data on national occupational demand (via NOICC*) interfaced with' 
comparable information in each state (via SOICC**) 

o Meta-evaluation reports from each state advisory council on vocational 
education 

o The independent findings from the several evaluations conducted or 
commissioned by the^fckional Institute of Education 

o Selected national evaluations of vocational education cc^iducted by the 
National Adyisory Council for Atocatiqrval Education 

< 

o Annual accountability reports from all states and territories 

o State plans with quantified goals from all states along with comments 
incorporated from the public hearings held in each state 

Braybrooke and Lindblom'S (1963) first proposition about the synoptic 
ideal is that it is not adapted to man's limited probiem-solving capacities. 
They elaborate on this limitation as follows: 

The -synoptic ideal not only fails to incorporate simplifying strategies 
but compounds the analyst's difficulties* by insisting on comprehensiveness 
of analysis., T t o insist on comprehensiveness is to rule out at the start 
many techniques for .simplification, t since omission is a chief 'principle 
of simplification, (p. 50) 

Strands of various synoptic approaches may tyegin quite mQdestly, but they 
appear to increase to the point of confusion. The past twenty years of fed- 
eral prescriptions, for evaluation and vocational education's response to such 
requirements are not unlike this. * ^ m 

' ' - *• • 

The proposition to be offered about the -utility of synoptic approaches to 

providing information ffom evaluation for program improvements i n vpcational 

education follows: 
S 



^National Occupational Information Coordinating Committee 
**State Occupational Information Coordinating Committee 
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< o Proposition 6 , "The chances of utilizing all existing evaluation 

findings about vocational education decrease in proportion to appro- 
priate decision-making groups 1 ability to — 

- perceive a need for them, 

- obtain them, 

- accept them, • >v 

- understand them as a coherent whole, . 

- reconcile their implications with the values they hold, 

- act on them through synoptic policy decisions. 

w inter active Problem Solving 

The rational ideal carries with it an assumption that problems are to be m 
solved through thought and analysis — preferably based on the verified finding^ 
of evaluation reports. Yet this view of the decision-making program improve- 
ment process ignores the many situations in which social interaction actually 
displaces the so-called analytic problem-solving process. Some alternatives 
to the tnse of professional social inquiry in social problem solving are exemp- 
lified by public hearings, the committee process, parliamentary procedures', ♦ 
and elections. 

Many of these interactions are ceremonies in which the rules of action are 
prescribed and scrupulously followed. Adherence to such rules may give the 
appearance of analytic, "rational" behavior. But they often mask — and provide 
rationalizations for — aonanalytic behavior. The program^ review process in 
vocational education may well be an example of such a cetemony. The proce- 
dures for conducting program reviews are lengthy and detailed, especially in 
those states where the process is most well developed. In addition, much of 
the process involves the review of masses of data, and the judgments of the 
reviewers are frequently combined into quantified entries on various £orms. 
The forms themselves can be aggregated, synthesized, and partitioned in ways 
that yield elaborate displays of data. The process and the data certainly 
appear to be analytic and rational; here is the relationship between profes- 
sional social inquiry and program improvement in action. 

The alternati¥e explanation — afforded by beginning^ with different 
assumptions — is that this is action (social interaction) in progress in a form 
that functions as a ceremonial substitute for an analytic process. The focus 
of the reviews is on standards and /criteria. There are often hundreds of 
them, typically clustered into areas such as scope and content of programs, 
planning and evaluation processes, quality and extent of ancillary reviews, 
adequacy of personnel, program management, use of other community resources, 
and sa oa. 

The Origins of the standards and criteria — which usually appear in the 
form of Guidelines and checklists — are assertions about program quality that 
are presumed to have a relationship with evidence of quality. Thus, if such 
an itemjig/**clarity of instructional objectives" appears (and some version of 
it almost always does), an assumption — not evidence — establishes a relation- 
ship between the^egree of clarity and some external index of quality such as 
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proficiency in performing some job* Ironically, those items having to do with 
obtaining sound .evaluation data on external quality indicators — such as 
followrup studies — have the same presumed relationship to program quality.* It 
is assumed that districts or states that have good follow-up systems should 
have better programs* It would be strange, indeed, to assume that the rela- 
tionship between the two is spurious, even inverse* 

m The point is that the program review process, which is the most prevalent 
form of evaluation practiced in vocational education (Smith et al., 1979) may 
well be $n effective technique for promoting program improvement. It is often 
^n intensive effort involving many well- in formed people .who are interested in 
improving programs* But the reason* for programs being improved through this 
process are far more likely to be derived from the process itself (a process 
of social interaction) than from the apparent rationale of using findings* from 
professional social inquiry for making rational, analytic program improvement 
decisions. # 

/The review process is essentially a tautology: Quality programs are those 
that have these presupposed characteristics; therefore, programs judged to 
have these- characteristics are quality programs. If programs are found 
wanting on some of the characteristics and the administrators are correcting 
some of the "deficiencies," they are, in fact, improving programs. Through- • 
out the program review process, the strict adherence to rules, procedures, 
checklists, report formats, and tffe J.ike permits the entire process to become 
a substitute for analytic problem solving. This is not to siiggest that prob- 
lem solving is not occurring; it may well be, and it may be muchjmore effec- 
tive than that which might be brought about by professional social inquiry. 

Similar examples of social irfter actions serving as substitutes for ana- 
lytic behavior might be seen (1) in the flow and submission of plans, and 
accountability reports from the local level through state to federal levels, 
(2) in the use of public hearings, (3) in the requirements of the NO ICC- and 
S0ICC3, (4) in the deliberations of state or local advisory committees, or (5) 
in carrying out meta-evaluations of vocational education's evaluations. . In 
each case, it may well be "that the v^fcious forms Of interaction among people, 
in which what they do, rather than what they think, or anyone else thinks 
about that problem moves toward the solution or preferred situation" (Lindblom 
and Cohen 1979, p. 20). 

Lindblom and ColJen (1979), in making the point that professional social 
inquiry often fails to recognize the problem-solving function of social inter- 
action, concede that the real issue, and the real challenge to professional 
social inquiry, is the possibility of a complementary relationship between the 
two modes of social problem solving. Problems are created when the assump- 
tions of professional social inquiry force one to relegate interactive problem 
solving to a realm of less sophistication and effectiveness than that of 
rational, analytic problem solving. 

The problem is further confounded when interactive processes are dis- 
guised as analytic ones. In these cases, we are misleading ourselves and 
others into thinking that our ceremonies are something other than ceremonies, 
t*hat ceremonies are necessarily ineffectual as social problem solving 
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processes, or that "sound" problem solving should always have the authorita- 
tive ring of professional sooial inquiry associated with it. This proposition 
is offered as an alternative to the set of assumptions we typically employ to 
justify the role of professional social inquiry in social problem „ solving: 

t 

o Proposition 7 . The consequences that steia fr&m najianalytic social 
interactions within vocational education's environment are far more 
likely to affect programs than current evaluation practices. Problems 
are created when — - 

* assumptions about proper problem-solving behaviors force a rejection 
of the, foregoing assertion. 

- non&nalytic social interactions are viewed or -portrayed as 'instances 
of professional social inquiry. 

- positive effects on programs are assuped to be more likely if based on 
valid evaluation results^. * 



Pursuit of the "Decision Maker " 

Since evaluations are intended to improve decisions, one mCist assume the 
existence of some identifiable individual or group as the decision-makipg 
recipient of the evaluations. The mechanistic metaphor that undergirds much 
of the evaluation activity in education forces this assumption and thereby 
creates yet another problem for the evaluation-program improvement relation- 
ship. For example, the data that are displayed through various gauges and 
printouts for those decision makers operating machines — from airplanes to gar- 
bage trucks — have a direct and usually unequivocal meaning for those known 
decision makers. 

* 

This analogy can be applied to the operators of our educational "systems" 
who must make decisions about the systems' "inputs/* "prbcesses," and "out- 
puts" in order to monitor,, adjust, recycle,. or restructure as necessary to 
maintain an efficacious input-output relationship. Our assumptions insist 
that decision makers need such information and that these individuals can be 
identified. 



The extent to which this perception is erroneous is seen most clearly in 
attempts to establish, after the fact, the identification of decision 
makers — especially in instances where the deci^on consequences are seen as 
negative. Beyond the simple "buck-passing" syndrome, it is often true that it 
is virtually impossible to ascertain the locus of decisions in education. The 
intertwined collage of prescriptions, report flows, and checks and balances 
among the local, state, and federal levels of jurisdiction in vocational edu- 
cation compounds the problem of identifying decision makers. . ' 

The assumptions that decision makers have been identified before the fact, 
and that there is need for evaluative information are particularly tenuous in 
vocational education. The amount of administrative turnover at all levels 
adds to the problem. The pattern at the federal level is such that it has 
been terme'd the "cyclical amnesia" syndrome by observers in Washington, D.Q. 
An additional issue is the misperceived, but apparently widely held, 

23 



Er|c 31 



asAimption that the agency or individual requiring or requesting an evaluation 
is the one who will use it. Even more naive is the assumption that the rea- 
sons expressed by decision makers for completing evaluations necessarily bear 
any relationship to the real reasons they are conducted. 

Weiss (1977) comments as follows about assumptions regarding decision 
makers* 

Almost all discussions of the use of research in policy t making start from 
the premise that at some discernible place and time policy gets "made," 
that there are people who see themselves and are seen by others as deci- 
sion "makers," and that singly and collectively they make arrangements to 
sodve a problem. The assumption that there are identifiable actors who 
mike a policy sometimes contradicts reality. 

In some circumstances, locating any people who are charged with respon- 
sibility for making a decision is difficult. Different organizations have 
Very different decision structures, and in some, the divisions of respon- 
m ^ sibility a*e sliced so fine that there seem to be nobody and no group 
with sufficient authority to move. ... In such amorphous and \dif fuse 
decision processes, the use of social Research is equally imperceptible, 
(p. 11) 

The following proposition is offered about the decision-based orientation 
of vocational education evaluation: 

o Proposition 8 . The probability of identifying a decision •maker or 
decision-making body in vocational education that has needed and acted 
upon evaluation findings for program improvement is decreased to the 
extent that— 

- the decision makers requesting an evaluation are not those intending 
to use it, • 

- the decision makers requesting an ^valuation are no longer in those 
roles at the completion of an evaluation, 

- authority for program improvement* is dissipated across several groups 
and levels of jurisdiction 

- the motives for an evaluation are not expressed by the purposes 
claimed for an evaluation, 

- the decision making process is based on ordinary knowledge anjd/or a 
social interaction process. 

Primacy of Goals and Organizational Efficiency / 

There are several other assumptions necessary to accept a relationship 
between evaluation and program improvement. But the realities required for 
these assumptions to be f&feusible and those that actually exist are often more 
than just misaligned a bit and reconciled by compromise; they are instead fun- 
damentally incompatible. To accept such assumptions and conduct evaluations 
accordingly is to run head on into their contrasts to reality. Our, notions 
about goals and organizational efficiency represent the two most widely held 
of such assumptions. 
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Logic demands that programs be evaluated in relation to explicit goals and 
that rational improvement decisions must narrow the/fap between the evidence 
and the goal* Assumptions about the primacy of goals constitute — especially 
in vocational education — the most deeply held, and least malleable, of all 
assumptions. A close second is the related assumption about the necessity of 
efficacious organizational behavior* Taken together, these assumptions form 
the rationale for most evaluation efforts: Because organizations are purpose- 
ful, goal-seeking entities that structure themselves in ways that permit them 
to seek out evidence of their progress toward their goals and make efficient 
corrections as necessary, it follows naturally that evaluations will focus on 
progress toward goals and provide the needed feedback, and" that the Organiza- 
tion will adjust its behavior accordingly* The type of organization best able 
to do this would have the following classic characteristics (McKibbin 1981): 

T 

o A hierarchical organizational structure that systematically orders 
communication and authority among formally established positions — An 
employee would report to only one supervisor. Authority would be 
commensurate with responsibility. , 

o Division of labor based on functional specialization — Employees would be 
* assigned to areas of specialization. 

0 A system of procedures, rules, and regulations covering the rights and 
duties of employees in work situations — Rules and regulations would 
avoid confusion and misunderstanding. 

o Promotion and selection based on technical competence — Rewards would be 
based upon performance. 

o Rational, systematic, goal-oriented organizational processes— *Specifying 
objectives improves performance. Organizations exist to attain 
specified goals, (pp. A, 12) 

Similarly, the goals to which the efficient organization aspires should have 
the following/ characteristics. (Clark 1980, p. 2). 

o Explicit . Goals are expressed and formally sanctioned by the 
organization. 

o Consentient . Goals are understood and shared by all organizational 
participants; singly and in subgroups. 

o Substantive . Goals focus on organizational products. 

o Nomotl>etic . Goals are dominated by institutional aspirations. ~ 

y . * * 

o. Concrete . A goal is operational in its definition; it can be employed 
effectively in defining organizational evaluation. 

o Certain. Goals are posited with confidence. 
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Each of these conditions is usually thought of as facilitating program 
improvement through evaluation. Certainly, at the extreme, it* is difficult to 
imagine evaluations leading to improvements in organizations that are dis- 
organized and do not aspire to known goals. 

Clark (1981) provides the following reasons for questioning whether many 
of the organizational characteristics listed apply to educational 
organizations. 

Educational organizations are archetypal, loosely coupled systems. If 
you examine your school, you are sure to find examples of (1) independence 
rather than Interdependence among units; (2) processes that seem discon- 
nected rather than linked; (3) actions isolated from consequences; and (4) 
individuals who function with little or no supervision. You will also 
discover organizational^ participants who are oblivious to these loose 
couplings and decouplings, or who deny their existence, (p. 33) 

) - 

These notions suggest new assumptions about educational organizations: 
"Maybe they all are really like that, not just the ones I've known"; and "Do 
^we forever gr-ant exceptions to excuse the realities around us or might we 
begin to suggest that those exceptions are the realities?" Also, 2flflbuld we 
continue to pursue ideals whose origins are much less clear to uj/Than the 
evidence of their futility?" 

Although this viewpoint does npt provide justification for summarily 
rejecting many of our traditional assumptions about the primacy of goals and 
organizational efficiency as conditions facilitative of program improvement 
based on evaluations, it does provide a basis for speculating on -some of the 
contrasts between assumptions and reality that might exist in vocational edu- 
cation. Consider, for example, the lists of possible contrasts provided in 
table 1. 

In light of "the foregoing, the following proposition is offered as a final 
speculation about conditions in vocational education — and, as with the other 
eight propositions, in any "other social program — which further mitigate the 
possibility of evaluations serving to improve programs. 

o Proposition 9 . Traditional assumptions about the primacy of explicit 
organizational goals and efficiency in pursuing such goals' on the basis 
of evaluative feedback constitute an anachronism based on an inaccurate 
perception of vocational education's processes, structures, and func- 
tions. To the extent that these assumptions continue to be accepted, 
the* chances of positive adaptations to future environments diminish. 



TABLE 1 



CONTRASTING ASSUMPTIONS AND PROBABLE REALITIES 



Condition 


Assumption 


1 Probable Reality 


Learner Intent, 
* | 


Clearly, vocational educa- 
tion's ovetall goal is 
occupational preparation. 


1 Less than one^h^lf of all 
I vocational education enroll- 
I ments are in programs of 
I occupational preparation. 


< i 
Relationships | 

\ ! 


The relationships between 
federal, state, and local 
levels % of vocational educa- 
tion are reasonably clear 
and * there is accountability. 


1 The locus of control for the 
1 direction $nd improvement 
I of vocational education is 
I at the local level — vested 
I principally in instructors, 
I most of whom are unaware of 
I or disinterested in federal 
I intents and prescriptions. 


Evaluation 
Incentives | 

i «• I 


Local agencies, supported 
by state departments, want 
and need evaluations for 
program improvement. As 
goal-seeking organizations, 
they are structured' to seek 
and act on such feedback. 
* 


I * 

I For many local agencies, 
I evaluations are necessary 
I evils* to. be tolerated and 
I dispatched as quickly as 
I possible. They see no rela- 
I tionship between evaluations 
| and program improvement. 
I Disincentives to conducting 
| rigorous evaluations 
I actually exist. 


Basis of | 
Content | 
Selection * | 
Decisions I 


The most important decisions 
ate those that determine 
content. • Thus, the best 
predictor of those decisions 
is empirical evidence of the 
content's fidelity to job 
requirements v 


I The best predictor of con- 
1 tent is a knowledge of past 
I traditions^ the interests 
I &f tenured instructors, and 
I the type of facilities and 
I equipment available. 


Linkages | 

4 ^ | 


The^array of actors, agen- 
cies, prerogatives, $nd 
responsibilities in voca- 
tional education is comple- 
mentary and systematic — all 
contributing clearly to 
improved learning 
oppor tun i ties. 

4 ■■- 


I The most "accurate" depic- 
I tion of the collective con- 
1 sequences from existing 
I linking agencies is that of 
I an "organized anarchy." % 
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ALTERNATIVES AND OTHER AGENDA 



The principal purpose of this paper will have been accomplished if 
.thoughtful readers responsible for evaluation or program improvement in voca- 
tional education reexamine their assumptions about the evaluation-program 
improvement relationship. It is hoped that such reexamination will be the 
catalyst necessary to bringing about constructive change. 

The premise of this paper is, As evaluation is presently practiced in 
vocational education, there is little discernable evidence that it contri- 
butes to program improvement; in fact, in many cases evaluations may well be 
^counter productive to prog#am improvement . Nine propositions have been offered 
4in support oFThis premise. They are structured around limitations in capa- 
city, validity, inter pretability, and utility. Thus, the argument has been 
set forth that evaluations do not presently contribute greatly to program 
improvement. The obvious question then is, "What should be done if program 
improvement efforts are to be more effective?" 



The "Stop!" Alternative 

* 

The first answer is to stop pursuing evaluation festivities that are 
purported to contribute to improvement but that, in reality, do not. The 
alternative is, if suclj evaluations, are to continue, the labeling of them as 
program improvement enterprises should cease. 

In the case of the first prescription, it is suggested that most eval- 
uation act^yities would cease if that approach were followed. In the second 
case, it is suggested that we would discontinue program improvement rhetoric. 
We would instead classify evaluations in ways more accurately (and— it would 
be argued here — more ethically) aligned with the functions they fulfill. Note 
that the functions they fulfill do not necessarily have any relationship to 
the purposes expressed to legitimize them. 

The first response will do much in and of itself to permit program 
improvement to become freed from the evaluation enterprises we claim as neces- 
sary to its progress. Similarly, evaluations that fulfill other functions 
(some of which are legitimate — perhaps even necessary) can continue about 
their business of fulfilling these functions unfettered by the expectation f 
that they contribute to program improvement. Consider for example, the fol- 
lowing possible functions (not purposes) and speculate on where we might place 
many of our existing evaluations and how they migjit change if we were to so 
classify them. 
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Advocacy 

Evaluation is seen as a tool (in some cases, a necessary one) to be u&ed 
by agencies and organizations having the responsibility for documenting their 
progress, accomplishments, and problems in fulfilling the purposes various 
publics hold for them. Such evaluations contribute directly to the content of 
reports and presentations such as- those made by — 

o United States - presidents, governors, mayors, or university officers in 
their annual "state of the union, state, city, *or university" 
addresses; 

o professional organizations or lobbies in education; 

o reports to stockholders; * » > 

o s annual reports on the accomplishments of vocational education. 



Compliance as Means 

Data and findings are supplied to others as a means for pursuing desired 
directions. Examples in this category include providing plans, accountability 
reports, and other completed forms to federal or staite offices. Evaluations 
used as means to ensure compliance serve a screening function. Those who wish 
to comply will do sp, prompted by a desire to enjoy positive or avoid negative 
sanctions that those requesting the information could bring to bear. Others 
will not. The pragmatic test used is whether the 9ost of providing the infor- 
mation is worth enjoying or avoiding such sanctions, there are twS ethical 
tests. One is whether the requesters nded such information to apply their 
sanctions fairly. The other is whether- those supplying the information can 
argue ,that any intentional inaccuracy in the information is warranted by the 
value of the desired sanctions they wish to pursue in relation to the per- 
ceived fairness by which the requesters apply sanctions. 



■ 'Ritual 

Ritual evaluations fulfill the function of signifying to interested pub- 
lics that various information exchanges (themselves rituals) present valued 
behaviors — the substance of which cannot be represented by information 
exchanges. Thus we have such things as . disarmament talks and treaties, and 
United Nations deliberations. We also have annual state plans, accountability 
reports, and justifications for distribution of funds. Ritual evaluations 
overlap with "compliance as. means" evaluations only td^the extent that infor- 
mation providers and requesters expect the information to represent past or 
planned behavior 4. 
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Adjuncts to Social Interaction or Ordinary Knowledge Problem Solving 

In this instance, evaluations fulfill the function of contributing — 
however modestly — to the processes by which human beings engage in "behavior 
that they believe to be more adaptive to their environment than previous 
behavior* To the extent that the findings from professional social inquiry 
supplement, confirm, or enrich the ordinary knowledge on which people base 
their behavior, or to the fextent that such findings assist them in finding 
ways to interact^ evaluations contribute to tne function of program 
improvement. 

) 

Other -Functions 

Other writers have^sdggested several additional reasons why evaluation 
tnight be used. They include using evaluation for such other functions as the 
following: 

o Evaluation is used to delay action; avoid taking responsibility for a 
decision; win kudos . . . discredit an ogponent or a disliked policy; / 

maintain nrPSfit>P! ItPPn fpufllnflt-nrcl c n 1 TOn f- • oor it a ae a f-fa'ln'lno r 



maintain prestige; keep [evaluators] soivlnt; serve as a training 
ground for [evaluators]; generate further research. (Weiss 1977, 
P. 15) 

0 Evaluation is used for setting realistic expectations; guiding policy; 
political value?. (Morell 1979, p. 244) 

o Evaluation is used to justify a federal role at all; to allocate funds; 
to meet proposal requirements; to prevent waste; to comply with 
regulations.' (Datta 1979, p. 67) 

The point of the "Stop!" alternative is simply to encourage us to ask what 
the consequences of evaluations really are and then either to label and pursue 
them accordingly or to discontinue those that are not worth completing. 



Content and Learner Selection Decisions 

The second Alternative focuses on content and learner selection decisions. 
The premise in this case is that the only decisions of consequence .in any 
area of education are those that determine what will' be learned and by whom, 
tn other words, the only ultimate standard that can be applied to determine 
the adequacy of education is that it is right and that it is fair. All other 
decisions and standards are subsidiary 'to these. 

In a conceptual framework related to. "employability institutions" (which 
include public vocational education), Walker and Pratzner (1981) portray all 
such institutions as being distinctive or adequate principally on the basis of 
the content they teach. The following excerpt from this framework is. pro- 
vided to explain further the position that all other tharacteristics jof such 
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institutions (in their nomenclature, processes, patterns, structures, or func- 
tions) follow from, and are justified only on the basis of, the opportunities 
they provide for teaching content. 

The following equation defines purposeful work socialization institutions 
(PWSI) as being a function (f) of their .capacity and abilities to provide 
opportunities to acquire various competencies (C 0 ) (.including the fairness 
of these equations) plus the frequency , intensity , and duration of attention 
they in fact give to those competencies (Cp, Cj,« Cj)). 



PWSI - f(C 0 , Cp, C l9 C D ) 

This view of PWSI forces us to look at all other attributes, conditions, 
and variables affecting PWSI as contributing to, or inhibiting, their capacity 
to teach content. If sight of this central view of PWSIs is lost, evaluators 
easily become diverted into looking at such things as process or pattern as 
ends in themselves. 

This formulation states that the competencies (C) actually addressed and, 
it is hoped, inculcated by PWSIs stem from the processes (PC) employed, the 
patterns (PT) these institutions form with their environment, the structures 
(S) by which they allocate resources and authority, and finallyj the functions 
(F) or range of consequences that accrue to the larger social unit. 



C = f(PC, PT, S, F) 

Profc esses (PC), then, result principally from different variations in 
instructional, techniques (IT), the ability of learning facilitators (LF A ), 
and the interest of the learners (Lj) exposed to those learning facilitators 
and instructional techniques. 



PC = f(IT, LF^Lj) 



Pattern (PT) refers to the set of interdependencies (Ij) established 
'with the environment surrounding a given PWSI or sets of PWSIs. These refer 
to the means by which an organization seeks support nourishment and avoids 
punishmetit from /its environment. Pattern is a function of the mix among 

incentives to establish such interdependencies (Ix) and disincentives (Dj) 
to do so. / * 



PT - f(Ix, Dj) 

The structure (S) of any organization refers to the configuration of 
rules, roles, and relationships by which it attempts to fulfill its purposes. 
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Structure results from the amount and continuity of resources (R A +C) (prin- 
cipally dollars) that support the institution. Note that amount and conti- 
nuity are important distinctions d.epicting the structure of a PWSI. Much of 
they»eemingly erratic behavior of many PWSIs (especially those with federal 
f *!^w^ n 8) can be attributed not just to the amount of resources available, but 
^Slsb.to uncertainties about their continuity. Further, structure is a conse- 
quence in part of the policies (PL) established by the organization. These . 
policies represent some translation of external requirements and internal 
needs. Finally, policy style (P s ) determines much of an organizations 
structure. This refers to important but elusive concepts such as leadership,, 
ambiance, and so forth. * 

s - f (r a +c; PL, P s ) 

Finally, function (F) is seen as the range of consequences that accrues 
from the existence of an organization. Whether positive or negative, latent 
or manifest, intended or unfR tended, immediate or long terii, these conse- 
quences and changes in those consequences stem from some feedback (f) or 
interpretation of social needs (/SN) as it is translated — however imperfectly — 
through the political support (js) of the PWSI. 



> F = f(SN, PS) ; > » 

Although the symbolism and terms of these ideas are not those typically 
used to depict vocatiortel education, the importance of opportunities to teach 
content is clear in their framework. Also, they establish the case for 
looking at everything else^ hat might be changed in vocational education as 
justifiable only on the basis of decisions related to the opportunity to teach 
content. This point is central to the program improvement alternative being 
proposed. It; means, for example, that it is relatively unimportant, at 
best^-at worst* damaging — for a vocational education* institution to decide on 
such things as the following unless such decisions can be justified, based on 
prior cbntent and learner selection decisions: 

o Hiring or firing instructors, administrators, evaluators, etc. 

o Building new f ac ili ties 

o Acquiring ixew equipment 

o Establishing linkages with whomever 

o Seeking additional funding sources 

o Forming new organizational structures 

o Conducting, evaluations of any type 
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o Changing curricula r ' j 

o Revamping philosophies 

o Changing teacher education or university graduate programs 

Content and learner selection decisions are those that determine what is 
to be taught and to whom , "Right" content selection decisions are those that 
provide students with an opportunity to acquire competencies that the publics 
to whom vocational education is ^responsible believe are necessary to increas- 
ing the satisfaction and productivity of t-he students' careers. " 

"Fair" learner selection decisions are those that permit "right" content 
opportunities to be provided to all students who will benefit from them. 
"Prografo improvement" then, is the process by which content and learner selec- 
tion decisions become more "right" and more "fair." Although these defini- 
tions reflect the familiar themes of "relevance" and "equity' 1 as hallmarks for 
improvement in vocational education, they are highlighted here as, consequences 
of content and learner selection decisions in order to reinforce the poiftt 
that these decisions have the following characteristic?. 

o They are made by d'esign or default by all vocational education institu- 
tions; they cannot not be made# 

\ 

o They supersede ,all other decisions; means will not create ends if the 
centrality of these decisions is td be reflected in practice. 

* . * 

o They provide the only logical bases for defending program improvement 
processes. 

^ o They will (consistent with the interpretabilit/ and utility propositions 
presented earlier) be made largely on the basis of ordinary knowledge* 
and a social interaction process of program improvement. 

' « ' 1 

If there is any hope of, evaluation serving program impxtovemptrt, functions, it 
must become a supplement to. these selection decisions* TTis* on this point 
that an optimistic scenario dan be developed pointing to a relationship ' 
between evaluation and progran improvement. The scenario requires these two 
conditions: \ 

» - 

1. ' The locus of content and learner selection decisions must be at the 

local district level; state and federal levels might .provide technical 
"assistance, but they cannot make these decisions. Their influence is 
t much like that of evaluation findings on program improvement; if they 
^ ate accepted, the most that can be hoped for i$ that their input will 
supplement* the ordinary knowledge and interactive processes on which, 
thesk decisions ar§ based i ; 

2. The content and learner selection decisions should be skplicit and 
• ^ publdci as should the processes by which they, were 'made. 



Both of these conditions must be genuinely met. If eithei? is seen or 
practiced as yet another compliance obligation, then all that will be gen- 
erated is more paperwork. It is not evident how local dis trie us 'might move 
toward these two Conditions. It seems safe to assume that ^if they choose to 
do so, they will find a way. Given the nature of the propositions set forth 
in this paper, to prescribe or itemize how they should do" so is both presump- 
tuous and contradictory. In its simplest form,, a local district might begin 
by providing a statement following a format such as the example provided in 
figure 2. This statement would accompany all courses and programs of instruc- 
tion offered in vocational education and would be made available to the vari- 
ous publics to whom vocational education is responsible (e.g., students, 
parents, other taxpayers, state and federal agencies, employers^, etc.). 

This example indicates the type of information to be made available. If 
conscientiously implemented, such anrjual content and student selection deci- 
sion reports could set in motion a positive process of program improvements 
based on the realities of both socistL problem solving and professional social 
inquiry. The positive scenario would contain the following elements: 

1. Legitimacy and autonomy wouid be afforded the local authority in 
making these decisions* Although some checks and balances would 
remain, and others would emerge over time, providing local districts 
with this right and responsibility removes much of the resentment — and 
consequent patterns of evasion — betweeri local districts and various 

y state, regional, and federal agencies. 

2. The* ff we-they ,, relationship between general education and vocational 
education at the 'local level might become less, strained in the pro- 
cess. The local vpcational administrator and the vocational staff te 
would become ceritral actors in the local decision-making process. 
They would be viewed less as .extensions of state or federal agencies 
making such decisions largely apart from — and naturally resented by — 
other local administrators and policy groups. ^ 

3. The selection decision statements, being explicit and* public, would 
cohstitute a tacit invitation for public response. If the statements 1 
rationales are suspec t, ob jec tionable, illogical — even irrational — 
in the minds of these publics, one must assume that those concerns 
would be expressed, heard, and eventually acted upon. If this assump- 
tion cannot be accepted, then grave doubts about the role of public 
education in a democracy might} exist. 

4. Local school personnel* inducing teachers, administrators, and 
boards, would — in anticipating these .explicit and public statements — 
place more importance on these selection decisions and eventually 
begin to see other decisions about their processes, patterns, struc- 
tures, or functions as subsidiary to, and justifiable by, their selec- 
tion decisions. In turn, these subsidiary decisions would become more 
understandable and defensible to both*' themselves and the public. 

5. The principal rationale for the initial selection decisions (perhaps 
for several years) would be based on ordinary knowledge and social 



4* 



\ 



FIGURE 2 

EXAMPLE OF A SIMPLIFIED STATEMENT 



CONTENT AND STUDENT DECISIONS: RATIONALE 

The , school district has decided to continue/ 

discontinue/expand/cut back its program of instruction 

in vocational education because: 

RATIONALE— ( S ta temen t of reason for above decision; could 
include precedent, history, opinion surveys, public meeting, 
instructors' views, administrative fiat, student demand, labor 
market surveys, expense' of facilities/equipment, default (no 
good reason not to), committee decisions, state department 
suggestions/requirements, federal suggestions/requirements, 
futures studies/scenarios, national-state-local occupational 
demand information* student testing, etc.) 

PROCESS — (The processes by which this decision was made; 
may be implied by rationale but could also include voting, 
computer analyses, public debate, advocacy hearings, program 
reviews, school board meetings, job/ task analyses, etc.) 

Further, because public education has the legal and moral responsi- 
bility to ensure that no group of students is denied the opportunity 
to enroll and succeed in this progtfin, these actions will be taken: 
(Recap* of decisions and commitments; might include recruitment cam- 
paigns, diagnostic testing, remedial instruction, tutoring, lottery 
enrollment system, compensatory enrollment system, follow-through 
♦seryices, etc.) 



\ 



Attac hments : mmm ^_ course syllabus 
* instructor resumes 



facility/equipment brochures 

synopses of evaluations, data syntheses, 
minutes of meetings, opinion surveys, 
committee reports, etc. 
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learning/ The principal process for arriving at the decisions — or 
pursuing social problem solving—would be through social interaction. 
Evaluation findings would be used as occasional adjunctS to these 
knowledge bases* 

As the process matures, public scrutiny and the genuine need to make the 
selection decisions as "right" and as "fair" as possible should lead to an 
increasing use of professional social inquiry as a means of justifying the 
decision^. Most importantly, the evaluations would originate from the real 
needs and motives of the individuals and groups who will use them. Perhaps 
for the first time, v the possibility exists that evaluations would be 
undertaken in which — 

o the expressed purposes coincided with the motives for the evaluations; 

or the purposes coincided with the functions of the evaluations; 

* 

"o the users of the evaluations were those requesting them; 

o the validity of the findings were more important than their nomative 
concurrence; 0 

o decision makers and evaluators expected the findings to supplement — not 
predict — subsequent decisions; 
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o the evaluations were disjointed and incr emental because the decisions 
they hoped to supplement were also disjointed and incremental; neither 
one is synoptic; 

* 

o evaluations could legitimately pursue a "discovery" mode- in which the 
extent of presumptive knowledge (or ignorance) of a theme, topic, or 
issue determines the evaluative tools and approaches to be employed — 
rather than the availability of tools determining the mode of inquiryV 
and therefore its focus and substance. < I 

The elements of the scenario outlined in this list are but skeletal pos- 
sibilities of what might emerge. However, this alternative is apparently the 
most viable. It is hoped that readers will consider its merits, reflect a bit 
on the realities of what they see around them, and ask whether they wish to 
continue to accept or express the same assumptions as before about evaluation 
and program improvement in vocational education. 
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Other Agenda 

Listed here in the form of questions are issues that warrant additional 
attention. The questions differ widely. The "decision rule" used to govern 
their inclusion is "when in doubt, do." The questions and occasional points 
of advice follow: 

1* How can vocational education credibly demonstrate its accomplishments 
and its limitations to the U.S. Congress ? The accountability syndrome 
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ha§ escalated in the past twenty years to a point where it now exists 
as a meta-reality with rules of its own far removed from once under- 
standable origins and purposes* A function of paper flow and compli- 
ance has replaced a purpose of information exchange and communication* 
Further escalation of this syndrome is the best prediction; "Stop!" is 
the best prescription* 

2. How can the "Is it worth it?" question be sensibly answered ? This 
question is both important and legitimate, especially at the federal 
level. It cannot be answered by using locally derived data. If the 
question warrants an answer, an independent evaluation should be con- 
ducted on the basis of careful and parsimonious national sampling with 
concerted attention given to standards of validity and longitudinal 
designs. With prudent use of the findings, we would expect equivo- 
cality and see the real question as, "Under what conditions and 
criteria might selected outcomes of vocational education correspond 

to the values various publics attach to it?" 

3 . Given all the possible limitations of professional social inquiry, 
what should be the role of educational research ? As idealistic (and 
perhaps, contradictory) as it might appear, educational research 
should be expanded. But it should be conducted under conditions 
that-- 

o are as apolitical as possible; 

o anticipate its findings to take at least a generation to be 

reflected in changed practices; the "research utilization" process 
is far more aptly seen from an ecological metaphor of gradual 
adaptation than a mechanistic metaphor of impact and utility; 

o insist upon as much discipline-based (behavioral/social science) 
research as possible. With the recommendation comes the humbling 
(and for some, antagonistic) recognition that "educational" research 
has no theoretical base and must borrow from ttose 4i sc ip^i nes that 
do. y« . 

4. ' How can tomorrow's vocational education leadership be prepared to 

make decisions which adapg better to the environments in which they 
will provide leadership ? The position taken in this paper is that 
they can best do so by learning to extract and examine assumptions — 
especially those that are unquestioned because of their prevalence 
, and tjie passion with Which they are held. To do this requires uni- 
versity graduate experiences fundamentally different from those that 
now exist. The theme of the needed differences could be seen in 
gradual shifts from certainty £o humility; from disdain for theory to 
centrality of theory; from evaluation to ptfof essional social inquiry; 
from program improvement to social problem solving; from mechanistic 
metaphors to ecological ones; and from a vocational versus liberal 
education stance to vocational education as a necessary means toward 
a liberal education. 
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The foregoing alternatives and other agenda have been offered in the 
spifit of constructive debate about evaluation and program improvement in 
vocational education. 

Evaluative activities exist to improve program improvement efforts and 
can/' in fact, do so. However, the Contribution of present practice in voca- 
tional education evaluation to program improvement is doubtful. The major 
obstacle is apparently the inability of practitioners to interpret and act 
appropriately on evaluative information. The time has come to pause and take 
a fresh look at the purposes of evaluation and the extent to which it is being 
implemented in service to those purposes, A second look at our assumptions 
about the relationship between evaluation and program improvement can become 
the springboard to constructive change. 
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